EFFECTS  OF  INCENTIVES  IN  ACQUISITION  COMPETITION 


1 


The  Limits  of  Competition  in  Defense  Acquisition 
Defense  Acquisition  University  Research  Symposium,  September  2012 

Keywords :  system  dynamics,  systems  thinking,  modeling,  incentives,  defense  acquisition, 
competition 


The  Effects  of  Incentives  in  Acquisition  Competition  on  Program  Outcomes 

William  E.  Novak 
Harry  L.  Levinson 
Carnegie  Mellon  University 
Software  Engineering  Institute 
June  1,  2012 


This  material  is  based  upon  work  funded  and  supported  by  the  Department  of  Defense 
under  Contract  No.  FA8721-05-C-0003  with  Carnegie  Mellon  University  for  the  operation  of  the 
Software  Engineering  Institute,  a  federally  funded  research  and  development  center. 

Correspondence  concerning  this  paper  should  be  addressed  to  William  Novak  or  Harry 
Levinson  at  the  Software  Engineering  Institute,  Carnegie  Mellon  University,  4500  Fifth  Avenue, 
Pittsburgh,  PA  15213. 


Contact:  wen@sei.cmu.edu  or  hll@sei.cmu.edu 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

01  JUN  2012  2' REPORT  TYPE 

3.  DATES  COVERED 

00-00-2012  to  00-00-2012 

4.  TITLE  AND  SUBTITLE 

The  Effects  of  Incentives  in  Acquisition  Competition  on  Program 

Outcomes 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROIECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Carnegie  Mellon  University, Software  Engineering 

Institute, Pittsburgh, PA, 15213 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

Presented  at  The  Limits  of  Competition  in  Defense  Acquisition,  Defense  Acquisition  University  Research 
Symposium,  18-19  Sep  2012,  Fort  Belvoir,  VA.  U.S.  Government  or  Federal  Rights  License 

14.  ABSTRACT 

A  review  of  acquisition  program  outcomes  would  make  it  appear  that  many  acquisition  programs  are 
destined  to  experience  recurring  schedule  slips  and  cost  overruns,  and  produce  poor  quality  systems.  We 
decry  these  circumstances,  but  the  acquisition  community  has  had  limited  success  in  correcting  them. 
However,  the  analysis  of  data  collected  from  assessments  of  many  software  acquisition  programs  has 
produced  insights  into  some  of  the  most  common  recurring  counter-productive  program  behaviors.  One 
result  of  this  research  has  been  the  identification  of  a  set  of  misaligned  incentives  that  are  a  significant 
force  in  driving  acquisition  programs  toward  poor  performance.  These  types  of  incentives  affect  all  aspects 
of  acquisition  programs,  including  the  competitive  forces  that  are  meant  to  reduce  acquisition  costs.  The 
system  for  acquiring  defense  systems  has  many  misaligned  incentives  that  unintentionally  promote 
program  outcomes  which  are  in  direct  conflict  with  the  stated  intent  of  the  acquisition  system.  These  occur 
in  all  phases  of  acquisition,  from  pre-award  to  sustainment.  Some  scenarios  illustrating  the  unintended 
consequences  of  acquisition  competition  include  ?  The  consolidation  of  multiple  needs  into  single  joint 
acquisition  programs  that  contractors  must  underbid  to  win  creates  schedule  pressure  that  drives  cost  and 
schedule  overruns,  and  encourages  stakeholder  programs  to  opt  out,  undermining  the  value  of  the  joint 
program.  ?  Short-term  performance  incentives  at  the  program  management  office  can  forestall 
sustainment  planning,  often  omitting  contractual  delivery  of  key  sustainment  tooling?and  as  a  result,  may 
inadvertently  lock  in  the  development  contractor  as  the  only  viable  choice  for  sustainment.  This  paper 
illustrates  each  of  these  dynamics  with  a  narrative  of  the  experience  of  an  acquisition  program,  drawn 
from  personal  interviews  with  the  program  office  and  contractor  staff.  The  paper  also  describes  ongoing 
research  studying  acquisition  incentive  issues  using  modeling  and  simulation  techniques.  Building  on  past 
work  with  qualitative  models  of  adverse  acquisition  dynamics,  we  are  building  executable  models  of 
program  behaviors  that  can  be  simulated,  validated,  and  analyzed  quantitatively.  This  research  approach 
is  described,  along  with  plans  to  model  candidate  solution  approaches  to  these  dynamics,  allowing  the 
proposed  solutions,  including  acquisition  system  policy  changes,  to  be  assessed  for  their  efficacy  in 
mitigating  or  resolving  the  problematic  behavior  before  a  candidate  policy  would  be  implemented. 


15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

Same  as 
Report  (SAR) 

18.  NUMBER 
OF  PAGES 

68 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


EFFECTS  OF  INCENTIVES  IN  ACQUISITION  COMPETITION 


2 


Abstract 

A  review  of  acquisition  program  outcomes  would  make  it  appear  that  many  acquisition 
programs  are  destined  to  experience  recurring  schedule  slips  and  cost  overruns,  and  produce  poor 
quality  systems.  We  decry  these  circumstances,  but  the  acquisition  community  has  had  limited 
success  in  correcting  them.  However,  the  analysis  of  data  collected  from  assessments  of  many 
software  acquisition  programs  has  produced  insights  into  some  of  the  most  common  recurring 
counter-productive  program  behaviors.  One  result  of  this  research  has  been  the  identification  of  a 
set  of  misaligned  incentives  that  are  a  significant  force  in  driving  acquisition  programs  toward 
poor  performance. 

These  types  of  incentives  affect  all  aspects  of  acquisition  programs,  including  the 
competitive  forces  that  are  meant  to  reduce  acquisition  costs.  The  system  for  acquiring  defense 
systems  has  many  misaligned  incentives  that  unintentionally  promote  program  outcomes  which 
are  in  direct  conflict  with  the  stated  intent  of  the  acquisition  system.  These  occur  in  all  phases  of 
acquisition,  from  pre-award  to  sustainment.  Some  scenarios  illustrating  the  unintended 
consequences  of  acquisition  competition  include: 

•  The  consolidation  of  multiple  needs  into  single  joint  acquisition  programs  that 
contractors  must  underbid  to  win  creates  schedule  pressure  that  drives  cost  and 
schedule  overmns,  and  encourages  stakeholder  programs  to  opt  out,  undermining  the 
value  of  the  joint  program. 

•  Short-term  performance  incentives  at  the  program  management  office  can  forestall 
sustainment  planning,  often  omitting  contractual  delivery  of  key  sustainment 
tooling — and  as  a  result,  may  inadvertently  lock  in  the  development  contractor  as  the 


only  viable  choice  for  sustainment. 
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This  paper  illustrates  each  of  these  dynamics  with  a  narrative  of  the  experience  of  an 
acquisition  program,  drawn  from  personal  interviews  with  the  program  office  and  contractor 
staff. 

The  paper  also  describes  ongoing  research  studying  acquisition  incentive  issues  using 
modeling  and  simulation  techniques.  Building  on  past  work  with  qualitative  models  of  adverse 
acquisition  dynamics,  we  are  building  executable  models  of  program  behaviors  that  can  be 
simulated,  validated,  and  analyzed  quantitatively.  This  research  approach  is  described,  along 
with  plans  to  model  candidate  solution  approaches  to  these  dynamics,  allowing  the  proposed 
solutions,  including  acquisition  system  policy  changes,  to  be  assessed  for  their  efficacy  in 
mitigating  or  resolving  the  problematic  behavior  before  a  candidate  policy  would  be 
implemented. 

Summary 

Misaligned  incentives  influence  many  aspects  of  acquisition  programs,  including 
competitive  forces,  in  ways  that  can  undermine  program  goals.  These  include  incentives  for  joint 
stakeholders  to  demand  custom  capabilities  from  joint  programs  and  thus  escalate  cost  and 
schedule,  as  well  as  incentives  acting  against  effective  sustainment  planning  that  can  lock  in 
contractors  for  sustainment  contracts.  The  paper  analyzes  these  instances,  and  discusses  ongoing 
work  using  modeling  and  simulation  to  analyze  the  effects  of  misaligned  incentives,  and  ways  of 


mitigating  their  effects. 
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The  Effects  of  Incentives  in  Acquisition  Competition  on  Program  Outcomes 

The  common  failure  of  acquisition  programs  to  achieve  their  desired  performance  in 
delivering  high-quality  systems  within  cost  and  schedule  constraints  ("DoD  acquisition 
outcomes,"  2005) — especially  those  developing  software -reliant  systems — is  an  all-too-common 
occurrence  in  modem  government  acquisition.  These  recurring  failures  have  a  direct  adverse 
impact  on  the  ability  of  the  U.S.  Department  of  Defense  to  be  able  to  support  the  warfighter  with 
the  systems  they  need,  and  that  the  U.S.  is  capable  of  providing.  Delayed  systems  withhold 
needed  capability,  and  wasted  resources  drain  budgets  that  could  be  used  to  develop  other  new 
systems. 

While  there  is  widespread  recognition  of  the  failures  in  acquisition  programs,  there  is  not 
general  consensus  as  to  the  reasons  for  this.  The  Defense  Acquisition  Performance  Assessment 
report  (Kadish,  2006)  acknowledges  that  the  underlying  causes  for  poor  performance  are  still  not 
well  understood — and  understanding  them  is  the  first  step  toward  improvement. 

The  SEI  is  regularly  asked  to  conduct  Independent  Technical  Assessments  (ITAs)  to 
investigate  the  reasons  why  specific  acquisition  programs  are  seriously  challenged,  and  even  fail, 
despite  the  best  efforts  by  government  and  industry  to  make  them  successful.  These 
investigations  have  provided  visibility  into  the  processes  and  forces  at  work  within  these 
programs.  Analysis  by  the  SEI  on  data  collected  from  over  100  ITAs  of  software-reliant 
acquisition  programs  has  produced  insights  into  some  of  the  most  common  ways  that  programs 
encounter  difficulties. 

We  know  that  acquisition  programs  do  not  fail  solely  for  technical  reasons.  One  set  of 
significant  reasons  why  acquisition  programs  may  substantially  exceed  budget,  overrun  schedule, 
and  ultimately  fail  may  be  due  to  organizational,  management,  and  cultural  issues  (Madachy, 
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2008,  Frangos,  1998).  In  the  SEI’s  direct  engagements  with  large  government  acquisition 
programs  we  have  seen  pervasive  recurring  failure  patterns  due  to  systemic  issues  in  the  form  of 
behavioral  dynamics  among  individuals  and  stakeholder  groups. 

Acquisition  Programs  as  Complex,  Dynamic  Systems 

The  focus  of  technical  acquisition  programs  is  most  commonly  on  the  complex  systems 
being  developed.  In  maintaining  that  focus,  however,  it  may  not  be  noticed  that  acquisition 
programs  themselves  are  complex,  dynamic  systems.  As  such  they  can  display  unpredictable  and 
even  seemingly  chaotic  behavior.  This  results  from  the  presence  of  feedback  between  the 
different  elements  of  the  organization,  producing  non-linear  behavior  that  defies  traditional 
mathematical  analysis.  The  complexity  of  feedback,  which  is  inherent  in  any  system  involving 
human  beings  who  can  spontaneously  interact,  coupled  with  time  delays  that  obscure  the 
relationships  between  cause  and  effect,  can  produce  this  unexpected  behavior  even  in  simple 
systems — which  means  that  actions  taken  may  have  unintended  side-effects  that  can  worsen  the 
problems  they  were  meant  to  solve.  What  is  important  to  realize,  however,  is  that  beneath  that 
complexity  there  are  recognizable  recurring  structures  that  drive  these  behaviors,  which  can  be 
understood  and  manipulated.  Past  work  done  at  the  SEI  in  developing  the  “Acquisition 
Archetypes”  (Novak  &  Levine,  2010;  McNew,  201 1)1  has  described  such  recurring  patterns  of 
counter-productive  organizational  behavior  based  on  observations  made  on  actual  acquisition 
programs. 

Once  we  recognize  that  an  individual  acquisition  program  (and  the  entire  acquisition 
system)  is  a  complex,  dynamic  system,  we  can  apply  appropriate  methods  and  tools  to  analyze 


1  Also,  Novak,  William  E.  &  Williams,  Ray  C.  An  Analysis  of  Recurring  Issues  Found  Across  12  U.S.  Air 
Force  Software-Reliant  Acquisition  Programs,  awaiting  publication. 
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and  investigate  the  impact  that  forces  such  as  competition  might  have  if  introduced.  Through  this 
analysis  process  we  can  investigate  the  unintended  consequences  that  can  result  from  the  use  of 
competition  in  acquisition.  Some  examples  of  consequences  that  affect  government  acquisition 
are: 

•  Vendor  “Lock-In  Competition  creates  an  incentive  for  companies  to  differentiate  their 
products  and  systems  from  existing  “standards,”  often  through  the  use  of  desirable 
vendor- specific  extensions.  Standards  are  inherently  commodities,  and  thus  for  a  vendor, 
standard  adherence  tends  to  drive  down  price,  which  becomes  the  primary  differentiator. 
The  incentive  to  produce  custom  extensions  is  to  gain  competitive  advantage  and  price 
differentiation  by  using  the  extensions  both  to  show  superior  capability,  and  also  to  create 
a  “switching  barrier”  to  “lock  in”  users  by  making  the  cost  and  effort  to  switch  to  a 
competing  product  high. 

•  Underbidding :  Underbidding  on  proposals  is  a  common  result  of  misaligned  incentives 
among  defense  contractors  that  is  continually  reinforced.  When  contractors  are  able  to 
win  contract  awards  through  underbidding  and  then  generate  additional  revenue  later  in 
the  program  to  recover  the  “lost”  revenue  from  the  underbid,  this  encourages  other 
contractors  to  employ  the  practice  to  an  even  greater  degree  to  remain  competitive  and 
win  the  next  contract. 

Structural  Dynamics  and  Misaligned  Incentives 

In  addressing  these  types  of  issues  the  Defense  Acquisition  Performance  Assessment 
report  (Kadish,  2006)  states  that  in  the  defense  acquisition  system  “Incentives  are  misaligned  - 
PMs  and  contractors  are  not  necessarily  rewarded  for  decisions  that  lead  to  lower  life  cycle  costs 
or  provide  a  better  balance  between  cost  and  performance.”  The  report  continues  on  to  claim  that 
“Efforts  to  reform  a  system  in  an  organization  as  large  and  complex  as  DoD  must  understand  and 
address  the  root  systemic  causes  of  organizational  and  individual  behaviors  to  be  successful.” 

Some  of  the  work  done  at  the  SEI  has  concluded  that  the  dynamic  behaviors  seen  in 
many  acquisition  programs  are  based  on  two  types  of  underlying  and  interlocking  forces: 
misaligned  incentives  and  structural  dynamics. 


EFFECTS  OF  INCENTIVES  IN  ACQUISITION  COMPETITION 


7 


Broadly  speaking,  misaligned  incentives  are  aspects  of  the  acquisition  system  such  as  its 
rules,  regulations,  policies,  and  guidelines  that  come  into  conflict  when  an  acquisition  decision¬ 
maker  must  trade-off  maximizing  the  achievement  of  one  objective  against  another.  For 
example,  misaligned  incentives  in  acquisition  programs  can  put  individual  or  program- specific 
interests  ahead  of  PEO  or  Service  interests,  turning  planned  cooperation  into  opposition,  and 
producing  poor  acquisition  outcomes. 

Structural  dynamics,  while  also  an  aspect  of  the  acquisition  system,  are  the  natural,  or 
even  in  some  sense  “physical”  aspects,  of  developing  and  acquiring  software-reliant  systems. 
These  structural  aspects  include  such  components  as  feedback  loops  and  time  delays.  For 
example,  diverting  developers  from  new  development  tasks  to  fix  defects  that  were  injected  as  a 
result  of  working  under  heavy  schedule  pressure  has  the  unintended,  yet  predictable  feedback 
consequence  of  slowing  development,  and  thus  further  increasing  schedule  pressure. 

Many  of  the  issues  caused  by  misaligned  incentives  in  acquisition  are  readily  apparent.  In 
one  example,  while  the  program  office  and  the  stakeholders  both  have  incentivizes  to  minimize 
program  schedule,  in  the  preferred  contracting  vehicle  for  unprecedented  development  (i.e., 
Cost-Plus  contracts)  the  contractor  has  an  implicit  incentive  to  maximize  schedule  in  order  to 
increase  revenue  and  maintain  a  stable  set  of  employees.  In  another  example,  when  it  comes  to 
requirements,  the  program  management  office  (PMO)  wants  to  minimize  requirements  changes, 
while  stakeholders  (users)  want  to  obtain  all  of  the  features  they  desire  (regardless  of  whether 
they  are  in  the  original  baseline),  and  the  contractor  (again  in  a  Cost-Plus  context)  is  happy  to  see 
the  revenue  benefits  of  both  the  additional  new  work,  as  well  as  the  rework  from  requirements 


changes. 


EFFECTS  OF  INCENTIVES  IN  ACQUISITION  COMPETITION 


8 


As  an  example  of  the  combination  of  misaligned  incentives  and  structural  dynamics,  we 
can  think  of  the  situation  in  which  strong  schedule  pressure  is  exerted  upon  a  software 
development  manager  by  higher-level  management  and  other  stakeholders.  The  amount  of  that 
pressure  that  the  manager  then  chooses  to  place  upon  the  developers  is  a  management  decision 
that  he  or  she  makes  based  largely  upon  incentives  that  are  often  misaligned  with  the  goals  of  the 
project:  the  potential  effects  of  late  system  delivery  on  his  or  her  career,  the  expected  effects  of 
the  schedule  pressure  on  the  developer  productivity  and  morale,  and  so  on.  The  “physics”  of  the 
underlying  structural  dynamic  that  contains  both  feedback  and  a  time  delay  is:  “If  a  developer  is 
worked  too  hard,  his  or  her  productivity  will  drop  off,  and  they  may  eventually  leave.”  If  the 
manager  chooses  to  exert  substantial  schedule  pressure  on  the  developers,  the  schedule  pressure 
will  at  first  increase,  but  then  ultimately  degrade  productivity  as  developers  begin  to  “burn  out” 
and  possibly  leave  the  organization.  While  the  amount  of  pressure  that  is  placed  on  the 
developers  is  a  management  choice  that  is  the  result  of  incentives,  the  results  of  chronic  schedule 
pressure  on  the  developers  are  not  a  choice — they  are  predictable  and  almost  inevitable. 

Taken  together,  misaligned  incentives  coupled  with  structural  dynamics  will  drive 
decisions  that  seem  (and  may  be)  reasonable  at  an  individual  level,  but  may  also  have  unintended 
consequences.  The  intent  of  increasing  schedule  pressure  on  developers  may  be  to  raise  their 
productivity  to  meet  a  schedule — but  the  longer-term  consequence  is  likely  to  be  the  opposite. 
The  manager  may  not  be  aware  of  this  consequence — or  may  make  a  calculated  gamble  that  the 
personal  consequences  to  himself  or  herself  of  trying  to  accelerate  development  through 
increased  pressure  on  the  developers  will  only  become  problematic  after  the  next  key  deliverable 
is  made.  After  that  the  schedule  pressure  will  decrease,  and  thus  the  productivity  benefits  will  be 
gained  without  suffering  the  worst  of  the  potential  costs  of  developer  burn-out  and  turnover. 
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With  both  types  of  forces  present  in  many  different  acquisition  situations,  their 
interdependence  injects  additional  complexity  with  the  potential  for  significant  unintended 
results.  The  interactions  of  the  two  aspects  frequently  combine  to  produce  the  counter-intuitive 
and  counter-productive  outcomes  that  are  often  seen  in  acquisition. 

Systems  Thinking 

One  technique  that  is  useful  for  analyzing  complex  systems  with  these  characteristics  is 
systems  thinking,  a  method  rooted  in  the  system  dynamics  work  pioneered  by  Dr.  Jay  W. 
Forrester  at  the  Massachusetts  Institute  of  Technology  (Forrester,  1971).  Systems  thinking  views 
systems  as  sets  of  components  that  have  complex  interrelations  occurring  between  them — and 
these  types  of  systems  occur  commonly  in  economics,  the  environment,  business,  politics,  and 
organizations  of  all  kinds.  A  key  tool  of  systems  thinking  used  in  this  paper  is  the  causal  loop 
diagram,  which  emphasizes  the  occurrence  of  feedback  loops  in  systems.  Causal  loop  diagrams 
illustrate  the  values  and  feedback  loops  present  in  a  system,  and  how  they  interact  with  and 
affect  one  another,  making  them  useful  as  an  explanatory  tool  to  present  the  sequence  of  events 
that  occurs  in  a  dynamic  behavior.  They  are  used  here  to  illustrate  the  structure  and  flow  of  the 
dynamics  in  two  example  acquisition  scenarios. 

This  paper  explores  ways  to  analyze  the  acquisition  system,  and  methods  to  investigate 
the  impact  of  competition  on  it.  Using  methods  such  as  systems  thinking,  the  presence  of 
misaligned  incentives  and  structural  dynamics  can  be  identified  and  analyzed  for  influences  they 
have  on  the  effectiveness  of  competition  in  acquisition.  This  paper  will  explore  some  pertinent 
examples  of  how  this  this  modeling  can  be  done  and  how  competition,  or  the  lack  thereof,  might 


play  out  in  these  scenarios. 
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Acquisition  Competition  Scenarios 

There  is  no  shortage  of  instances  in  which  competition  affects  the  performance  and 
outcome  of  acquisition  programs.  These  occur  in  all  phases  of  acquisition,  from  pre-award  to 
sustainment.  Two  useful  scenarios  illustrating  the  unintended  consequences  of  acquisition 
competition  are: 

•  The  consolidation  of  multiple  needs  into  single  joint  acquisition  programs  that 
contractors  must  underbid  to  win  creates  schedule  pressure  that  drives  cost  and 
schedule  overruns,  and  encourages  stakeholder  programs  to  opt  out,  undermining  the 
value  of  the  joint  program. 

•  Short-term  performance  incentives  at  the  PMO  can  forestall  sustainment  planning, 
often  omitting  contractual  delivery  of  key  sustainment  tooling — and  as  a  result,  may 
inadvertently  lock  in  the  development  contractor  as  the  only  viable  choice  for 
sustainment. 

The  following  sections  present  more  detailed  discussions  of  these  two  scenarios,  which 
have  been  adapted  from  actual  acquisition  programs.  While  no  specific  programs  or  individuals 
are  identified  so  as  to  preserve  confidentiality,  the  narratives  presented  are  based  on  interviews 
conducted  with  program  staff,  contractors,  sponsors,  and  others,  and  direct  quotes  are  used  when 
possible. 

Scenario  #1:  “Consolidation  into  Joint  Programs” 

The  first  scenario  presents  a  dynamic  that  involves  the  effect  of  the  larger  acquisition 
system  context  on  the  way  acquisition  programs  are  approached. 

“Consolidation  into  Joint  Programs”  Scenario  Narrative 
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The  program  was  started  in  a  context  of  a  tightening  defense  budget,  together  with  a 
growing  need  for  improved  interoperability  as  systems  were  becoming  more  interconnected.  One 
response  to  these  two  trends  was  an  increasing  interest  in  the  use  of  joint  programs  as  a  way  to 
partially  address  both  issues.  Defense  contractors  responded  to  the  shift  by  aggressively  pursuing 
the  joint  program  contracts,  correctly  recognizing  that  these  larger  efforts  would  be  especially 
important  in  an  environment  that  would  feature  fewer  programs  overall — and  aggressive  bids 
were  an  important  strategy  in  winning  the  contract  award. 

Off  and  Running 

The  concept  was  an  ambitious  one,  and  had  been  under  study  for  almost  a  decade  before 
the  Joint  Program  Office  (JPO)  was  stood  up.  The  new  capability  was  originally  conceived  as  a 
way  of  improving  interoperability  across  a  variety  of  platforms  from  the  different  Services  by 
building  a  single  capability  that  would  be  incorporated  into  each  platform.  This  also  solved  the 
commonly  understood  problem  that  “. .  .there  wasn't  enough  funding  for  the  Services  to  each 
develop  their  own.”  According  to  one  of  the  Service  stakeholders,  “They  decided  that  whatever 
they  did  would  have  to  be  done  by  all  of  the  Services.  They  didn’t  want  the  Services  to  all 
separately  implement  to  one  standard,  as  they  had  in  the  past — they  wanted  to  develop  the 
software  once,  and  make  sure  that  it  would  be  done  in  the  same  way  across  all  platforms  and 
Services.”  That  way,  the  thinking  went,  they  would  “. .  .pay  for  the  capability  only  once,  instead 
of  many  times”  to  avoid  having  “...50  disparate  [capabilities]  that  the  Services  were 
developing,”  with  the  problem  being  that  “. .  .you  not  only  pay  multiple  times,  but  you  get  poor 
interoperability  as  well.”  At  the  same  time,  the  officials  in  charge  “. .  .may  have  been  afraid  that 
if  they  gave  it  to  a  Service,  that  they  wouldn’t  follow  through  with  it”  and  produce  something 


that  would  be  generally  usable. 
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Still,  not  everyone  was  fully  on  board,  or  completely  sold  on  the  joint  concept.  There 
were  some  serious  implications  of  taking  this  approach  to  developing  the  new  capability.  It  was 
clear  to  each  of  the  participating  programs  that  .  .the  [stakeholder  program]  becomes  dependent 
on  the  group  doing  the  [joint]  capability — so  they’re  now  dependent  on  [that]  to  achieve  their 
own  requirements  for  their  platform.”  For  the  Army,  that  meant  they  were  now  “. .  .totally 
dependent  on  the  [joint  capability],”  and  “. .  .if  [it]  fails,  the  rest  of  the  house  of  cards  comes 
tumbling  down,  and  you’re  back  to  Service  capability.” 

As  a  senior  acquisition  executive  observed  later,  left  to  their  own  devices  “The  Navy 
would  have  gone  with  [their  legacy  system]”  and  “. . .the  Army  would  have  gone  grudgingly. . .” 
along  with  the  joint  effort.  It  was  common  knowledge  to  those  supporting  the  new  program  that 
“Without  a  joint  requirement,  it  significantly  increases  the  risk  of  a  stovepipe  approach”  because 
“It’s  easier  to  do  a  stovepipe  system  than  to  do  joint.”  As  one  senior  official  noted,  “People  don’t 
want  joint  programs — they  want  to  control  their  own  destiny.” 

The  Catch 

One  of  the  biggest  challenges  in  getting  a  joint  program  off  the  ground  is  defining  the 
scope  of  the  new  functionality  that  it  will  provide.  While  it  would  have  been  ideal  to  offer  a 
capability  that  would  not  only  be  the  union  of  the  legacy  system  features  that  each  of  the 
platforms  already  enjoyed,  but  which  would  also  add  to  that  the  new  capabilities  being 
developed,  budgetary  constraints  indicated  that  would  be  difficult.  For  just  such  reasons,  one  of 
the  joint  program  leads  acknowledged  that  “Joint  programs  have  always  been  focused  on 
providing  the  lowest  common  denominator”  across  stakeholders  in  terms  of  functionality.  This 
created  a  dilemma,  because  as  another  program  lead  commented,  “The  Services  are  insisting  that 
the  [new  software]  be  capable  of  performing  all  of  the  functions  that  their  legacy  systems  can  do 
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today. . .  They  won't  change  over  to  using  the  [new  software]  if  it  means  that  they  will  lose  any 
capability.”  Almost  as  soon  as  the  CDD  requirements  had  been  defined,  the  joint  program 
realized  that  “The  Services  want  more  in  the  system  than  what's  in  the  CDD.” 

Knuckling  Under 

When  the  stakeholder  programs  began  to  come  to  the  JPO  to  advocate  for  the  specific 
requirements  they  needed  that  were  outside  the  baseline  the  JPO  had  proposed,  the  JPO  had  a 
difficult  choice  to  make:  either  agree  to  add  multiple  sets  of  custom  requirements,  and  as  a  result 
slip  schedule,  drive  up  cost,  risk,  and  complexity,  and  thus  shake  the  stakeholders’  confidence  in 
the  JPO  to  the  point  where  many  stakeholder  programs  might  decide  to  leave  the  joint 
program — or  refuse  to  add  the  new  requirements,  and  by  doing  so  risk  angering  the  stakeholder 
programs  enough  that  they  might  leave  the  joint  program.  Either  option  could  lead  to  the  failure 
of  the  joint  program. 

Initially  the  JPO  was  reluctantly  willing  to  add  the  new  requirements  to  the  capability, 
preferring  to  avoid  the  immediate  wrath  of  the  stakeholder  programs  in  favor  of  the  longer-term 
cost  and  schedule  growth  issues.  According  to  a  senior  official,  “The  Services  meandered  with 
the  requirements  process,  and  the  JPO  allowed  it — they  should  have  held  the  line.”  It  was  clear 
that  the  stakeholder  programs  wouldn’t  be  satisfied  until  they  each  reached  the  same  level  of 
functionality  that  their  legacy  systems  had  provided,  and  that  was  going  to  require  extending  the 
requirements  baseline.  As  the  JPO  noted,  the  joint  program  was  “still  chasing  requirements  from 
the  Services  that  haven't  been  formally  required  or  requested.” 

When  a  new  colonel  took  over  as  PM  at  the  JPO  in  the  program’s  fourth  year,  he  resolved 
to  run  things  differently,  and  as  a  result  the  joint  program  “. .  .became  very  defensive  about  doing 
any  additional  work  [that  wasn’t  already  in  the  requirements],  because  doing  it  would  delay 
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them,  and  make  them  look  bad.”  Despite  the  fact  that  refusing  to  add  still  more  [customized] 
functionality  to  the  program  was  a  discipline  that  most  agreed  was  long  overdue,  “the  Services 
didn't  like  the  Colonel  holding  the  line  on. . .  functionality  and  funding.” 

Unintended  Consequences 

With  the  additional  requirements  that  had  been  added  to  the  baseline,  some  recognized 
that  the  JPO  now  had  “. .  .a  tough  engineering  problem,  and  no  one  appreciates  the  complexity, 
work,  risks. . .”  that  it  presented.  They  were  even  warned  by  a  technical  advisor  that  they  were 
“. .  .embarked  on  a  very  high  risk  software  venture.”  The  consequences  of  that  complexity  and 
risk  started  to  become  apparent  as  development  progressed,  and  one  senior  executive  noted  with 
growing  concern  that  “. .  .the  schedule  has  been  continually  slipping  over  time.  Every  time 
they’ve  gotten  close  to  delivery,  they’ve  slipped.”  The  program  cost  was  growing  accordingly, 
following  a  pattern  described  by  one  Service  stakeholder  as  “. .  .double  the  cost,  and  multiply  the 
schedule  by  a  factor  of  2-3,  and  [the  program]  fits  right  into  that  model.  The  schedule  slips,  and 
the  cost  grows.”  This  was  alarming  to  the  stakeholder  programs  who  were  “. .  .worried  about 
[the]  cost,  schedule  and  risk. . .  [of]  trying  to  deliver  a  combat  system”  when  “a  piece  of  this  is 
being  developed  by  an  outside  organization  [i.e.,  the  JPO]  with  a  track  record  of  sliding 
schedules  and  faulty  systems,  and  this  [capability]  is  going  into  the  center  of  this  combat 
system. . .”  To  put  it  simply,  “The  potential  for  [the  capability]  to  harm  [their  platform]  has  the 
Navy  scared.” 

Defecting 

As  the  schedule  continued  to  slip  and  costs  continued  to  rise,  a  key  Army  program  stated 
that  “Their  confidence  in  the. . .  [ability  of  the]  JPO  to  deliver  has  eroded  over  the  last  year  and  a 
half’  and  that  they  had  “very  little  confidence  [the  JPO]  will  be  able  to  deliver  what  they  say 
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they  will  in  the  time  they  say  they  can.”  The  JPO  noted  the  vagueness  of  the  “lack  of 
confidence”  charge,  with  one  acquisition  executive  suspecting  deeper  reasons  for  it,  noting  that 
“If. . .  confidence  were  their  issue,  they  could  push  for  more/better  testing — but  then  they  might 
have  to  implement  it.” 

Given  the  situation,  there  was  uneasiness  on  the  part  of  several  programs,  as  well  as  their 
Services.  One  program  reviewer  said  “I’m  not  sure  that  the  Navy  really  wants  to  do  this,”  and  a 
Navy  captain  acknowledged  that  “We  have  to  integrate  it  because  OSD  and  JROC  have  said  that 
we  have  to.”  Still,  the  Army  was  more  vocal  about  their  concerns.  They  had  made  it  clear  to  the 
JPO  that  “The  Army  wants  to  get  the  [joint]  requirements  out  of  their  CDD.  They  want  to  have  a 
way  out.”  With  the  amount  of  conflict  that  was  starting  to  show  at  the  review  meetings,  others 
began  to  wonder  if  the  current  approach  could  succeed.  A  member  of  a  team  that  reviewed  the 
joint  program  recommended  that  they  should  “Let  the  Army  interface  with  their  own  system,  and 
control  their  own  destiny,”  while  one  of  the  Service  stakeholders  believed  that  “If  the  Army  goes 
off  on  its  own,  everyone  loses.  Title  X  gives  the  Services  the  right  to  be  wrong,  and  they’ve 
certainly  exercised  that  right.” 

Unraveling 

Not  only  were  the  total  costs  of  the  program  increasing,  but  the  prospect  of  losing  one  or 
more  of  the  current  stakeholder  programs  meant  that  the  remaining  programs  would  have  to 
shoulder  that  additional  share  of  the  financial  burden.  As  it  was,  according  to  one  Service 
stakeholder,  “we  thought  it  would  be  $500M  originally,  and  then  it  got  blown  up  into  a  $1B 
program.  That’s  when  the  wheels  started  to  come  off,  and  the  Service  contributions  started 
having  to  ratchet  up.”  The  prospect  of  an  additional  cost  increase,  together  with  the  delivery 
issues,  was  stretching  the  remaining  stakeholders’  faith  in  the  effort.  As  one  program  reviewer 
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put  it,  “The  JPO  has  had  their  share  of  delivery  issues  in  the  past,  and  the  [stakeholder]  programs 
paid  the  price  for  that  in  the  past,  and  have  lost  confidence.”  Another  Service  stakeholder  was 
more  blunt  about  the  degree  of  conflict  that  was  starting  to  emerge  between  the  stakeholder 
programs  and  the  JPO,  saying  “People  get  spitting  mad,  and  are  shaking  with  rage  at  some  of  the 
meetings.” 

Collapsing 

Ultimately,  as  a  member  of  a  program  review  team  summed  it  up,  “The  key  question  is 
whether  the  Services  want  a  joint  solution. . .  or  do  they  want  to  go  their  own  way  and  perpetuate 
their  interoperability  challenges?”  One  of  the  stakeholder  program  managers  had  admitted  that 
the  joint  program  was  “. .  .a  train  wreck  in  progress,”  and  stated,  “I  believe  that  it  will  fail.”  In 
looking  at  the  level  of  hostility  that  had  erupted  between  the  stakeholder  programs  and  the  JPO, 
it  was  clear  to  one  of  the  acquisition  executives  that  “The  Services  don’t  want  this.”  In  fact,  one 
of  the  Service  stakeholders  had  recommended  that  the  DoD  should  “Kill  the  program  as  we 
know  it  today,  and  salvage  what  they  have.” 

The  words  of  that  stakeholder  proved  to  be  prophetic  when  it  was  formally  announced 
shortly  thereafter  that  the  program  had  been  cancelled.  Even  beyond  the  failure  of  this  specific 
program,  however,  was  that  despite  the  straightforward  economic  rationale  for  joint  programs, 
there  was  a  growing  realization  that  what  had  happened  was  “symbolic  of  a  deeper  problem.” 
Joint  programs  were  fundamentally  more  challenging  than  other  types,  and  this  program  was  just 
one  example  of  the  kinds  of  issues  they  would  continue  to  see  on  other  joint  programs.  This 
began  to  put  the  “needs  consolidation”  strategy  into  a  different,  and  much  less  favorable  light.  As 
a  senior  acquisition  executive  who  was  responsible  for  the  program,  and  who  had  made  a  strong 
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effort  to  force  the  Services  to  work  together,  asked  in  the  aftermath,  “What  else  would  you  have 
had  me  do?” 

“Consolidation  into  Joint  Programs”  Scenario  Analysis 

Joint  programs  have  become  common  throughout  the  DoD  as  a  way  of  both  reducing 
costs  and,  in  some  cases,  improving  interoperability.  According  to  former  Secretary  of  Defense 
Robert  Gates  .  .The  military's  operations  have  become  very  joint  -  and  impressively  so. . .” 
(Bennett,  2009).  However,  those  benefits  have  come  at  a  price,  and  have  given  the  concept  of  the 
joint  program  a  poor  reputation.  As  described  by  Kadish  (Kadish,  2006),  “Department  of 
Defense  acquisition  strategies  that  consolidate  multiple  capability  needs  into  ‘single  weapon 
system  procurement’  force  industry  into  ‘must  win’  cost  competitions.”  The  concept  of  a  “single 
weapon  system  procurement”  that  will  address  multiple  different  needs  has  come  in  many 
instances  to  imply  a  joint  program.  The  underbidding  that  is  frequently  employed  by  the 
competing  contractors  and  is  then  often  captured  in  the  contract  award  budget  creates  the  initial 
schedule  pressure  that  helps  to  undermine  the  joint  program  from  the  start.  The  inadequate 
funding  can  set  the  program  up  for  significant  cost  and  schedule  overruns — thus  creating 
opportunities  for  joint  program  participants  to  leave,  blaming  their  departure  on  rising  costs  and 
slipping  schedules,  undermining  the  value  of  the  program  (by  reducing  the  number  of 
participants),  and  potentially  causing  its  collapse. 

Following  the  systems  thinking  approach  of  creating  a  causal  loop  diagram  of  the 
dynamic  described  by  the  narrative  is  a  useful  way  of  simplifying  the  behavior  to  its  essential 
components.  Figure  1  depicts  the  basic  causal  interactions  and  flows. 
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Figure  1:  Overview  of  the  “Joint  Programs”  Dynamic 


To  read  the  diagram  as  a  story,  ideally  in  a  joint  program  as  the  Number  of  Stakeholders 
Interested  in  Joint  Capability  increases,  the  Amortized  Cost  of  Joint  Capability  per  Stakeholder 
decreases,  which  increases  the  Stakeholder  Confidence  in  Joint  Program  Performance,  and  feeds 
back  to  help  further  increase  the  Number  of  Stakeholders  Interested  in  Joint  Capability  (loop  R). 


2  Causal  loop  diagrams  depict  the  dynamic  causes  and  effects  in  a  situation  by  showing  how  variables 
(represented  by  nodes)  relate  to  and  influence  one  another  (represented  by  arrows).  The  effect  of  the  arrows  is 
labeled  “S”  for  “Same”  (i.e.,  when  one  variable  increases,  the  next  variable  increases  as  well — or  when  one  variable 
decreases,  so  does  the  other)  or  “O”  for  “Opposite”  (i.e.,  when  one  variable  goes  up,  the  next  one  declines,  or  vice 
versa).  The  arrows  often  come  together  to  form  feedback  loops,  which  are  labeled  either  “B”  for  “Balancing,” 
describing  loops  that  converge  toward  a  stable  equilibrium  (where  x  increases,  y  decreases),  or  “R”  for 
“Reinforcing,”  describing  loops  that  continually  or  even  exponentially  increase  or  decrease  (where  x  increases,  y 
increases).  The  term  “Delay”  on  an  arrow  indicates  actual  time  delays  that  occur.  Such  time  delays  are  significant  to 
a  human  being  attempting  to  control  a  system,  as  they  can  obstruct  the  ability  to  clearly  see  the  connections  in  cause- 
and-effect  relationships. 
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This  effect  helps  the  joint  program  to  attract  stakeholders.  However,  while  the  Number  of 
Stakeholders  Interested  in  Joint  Capability  grows,  the  Number  of  Custom  Capabilities 
Developed  also  grows,  driving  up  the  Joint  Capability  Complexity  and  Risk  (B2).  This  depresses 
Stakeholder  Confidence  in  Joint  Program  Performance  and  drives  down  the  Number  of 
Stakeholders  Interested  in  Joint  Capability.  Similarly,  as  the  Number  of  Custom  Capabilities 
Developed  grows,  it  also  drives  up  the  Schedule  and  Funding  Needed  to  Deploy  Joint  Capability, 
increasing  the  Gap  Between  Available  and  Needed  Schedule  and  Budget  (B3).  This  contributes  to 
Budget  and  Schedule  Overruns ,  and  reduces  Stakeholder  Confidence  in  Joint  Program 
Performance.  In  essence,  the  loops  on  the  right  side  of  the  diagram  are  driving  stakeholders 
away  from  the  joint  program. 

However,  external  to  the  joint  program  itself,  there  is  another,  larger  dynamic  playing  out 
in  loop  Bl.  As  the  Defense  Budget  decreases,  there  is  a  corresponding  increase  in  the 
Consolidation  of  Multiple  Needs  into  Joint  Programs  in  an  effort  to  reduce  costs.  This  leads  to  a 
decrease  in  the  Number  of  Other  Acquisition  Programs,  and  an  associated  increase  in  the 
Competitiveness  of  the  Bidding  Process.  This  in  turn  increases  the  use  of  Underbidding,  which 
reduces  the  Available  Schedule  and  Budget  for  Joint  Capability  when  the  contract  is  awarded. 
Just  like  the  internal  pressures  of  the  joint  program,  this  loop  also  increases  the  Gap  Between 
Available  and  Needed  Schedule  and  Budget,  again  causing  Budget  and  Schedule  Overruns, 
reducing  Stakeholder  Confidence  in  Joint  Program  Performance,  and  driving  down  the  Number 
of  Stakeholders  Interested  in  Joint  Capability.  This  then  increases  the  Gap  Between  Needed 
Stakeholders  and  Actual  Stakeholders  (as  compared  to  the  defined  Number  of  Stakeholders 
Needed  for  Joint  Program  Success),  increasing  the  Number  of  Joint  Program  Failures  (since  this 
is  a  common  way  for  joint  programs  to  collapse — when  they  no  longer  have  a  “critical  mass”  of 
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participating  stakeholder  programs),  and  ultimately  undermining  the  further  Consolidation  of 
Multiple  Needs  into  Joint  Programs. 

“Consolidation  into  Joint  Programs”  Scenario  Implications  for  Competition 

As  the  causal  loop  diagram  illustrates,  there  are  two  separate  dynamics  driving  the  joint 
program:  the  exogenous  dynamic  forming  the  loop  outside  of  the  acquisition  program  that  is 
driving  the  competitive  pressure  that  leads  to  underbidding  (B 1),  and  the  endogenous  dynamic 
that  is  an  inherent  property  of  the  social  dilemma  structure  of  a  joint  program  that  naturally 
tends  to  tear  it  apart  (loops  R,  B2,  and  B3).  In  the  exogenous  dynamic  the  well-intentioned  belief 
that  the  consolidation  of  multiple  similar  individual  needs  into  fewer  joint  programs  will  reduce 
costs  ends  up  creating  a  strong  incentive  for  underbidding,  creating  unrealistic  cost  and  schedule 
goals  that  can’t  be  met,  and  setting  up  the  joint  effort  for  failure,  thus  eventually  undermining  the 
entire  approach  of  consolidating  needs  into  joint  programs.  In  the  endogenous  dynamic  the 
problematic  reconciliation  of  disparate  joint  program  stakeholder  needs  into  a  single  solution, 
when  each  stakeholder  has  an  incentive  to  force  the  JPO  to  satisfy  their  requirements  at  the 
expense  of  the  program,  has  the  unintended  consequences  of  increased  cost,  schedule, 
complexity,  and  risk,  all  contributing  again  to  potential  program  failure  or  cancellation. 
Ultimately,  both  dynamics  work  to  drive  up  cost  and  schedule,  thus  undermining  confidence  and 
eventually  driving  stakeholders  away. 

It  should  also  be  noted  that  while  it  is  a  point  of  interest,  it  happens  that  the  underlying 
motivation  for  underbidding  is  not  relevant  to  the  behavior  of  the  exogenous  dynamic.  Whether 

3  A  social  dilemma  refers  to  a  class  of  problems  in  which  the  desires  of  individuals  to  gain  individual 
benefit  or  avoid  shared  costs  end  up  imposing  costs  on  the  others  around  them,  making  everybody  worse  off.  There 
is  a  paraphrased  saying  attributed  to  Adam  Smith  that  states  this  dynamic  as,  “ Individually  optimal  decisions  lead  to 
collectively  inferior  solutions.” 
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underbidding  occurs  due  to  honest  errors  in  estimation,  or  through  a  deliberate  attempt  to 
produce  a  low  bid  that  will  win  the  contract,  the  effect  is  the  same.  The  program  will  be  under 
cost  and  schedule  pressure  from  the  beginning,  which  will  undermine  its  performance.  However, 
it  should  be  noted  that  if  low  bids  were  primarily  due  to  honest  estimation  errors,  this  wouldn’t 
explain  why  underbidding  occurs  so  frequently  in  acquisition  programs — an  observation  which 
points  to  underbidding  as  an  intentional  response  to  the  incentives  of  the  acquisition  contracting 
process.  A  possible  explanation  for  the  prevalence  of  underbidding  in  acquisition  can  be  found  in 
George  Akerlof  s  (Akerlof,  1970)  Nobel  Prize  winning  work  on  “Gresham’s  Dynamic,”  a 
corollary  to  Gresham’s  Law4.  Akerlof  pointed  out  that  effective  but  dishonest  practices  in  any 
business  would,  over  time,  become  commonplace,  and  drive  out  more  honest  practices.  In  the 
same  way,  once  underbidding  is  employed  and  found  to  be  successful  in  winning  contracts, 
accurate  bidding  will  tend  to  become  scarce. 

Scenario  #2:  “Sacrificing  Sustainment” 

The  second  scenario  presents  a  dynamic  that  involves  the  sustainment  planning  aspect  of 
software-intensive  system  development. 

“Sacrificing  Sustainment”  Scenario  Narrative 

The  program  was  building  the  next  generation  of  a  high-speed  communication  system  to 
serve  a  variety  of  users,  and  was  composed  of  many  different  COTS  components  to  leverage 
commercial  technology.  As  the  contractor’s  project  director  described  it,  “The  nature  of  the 
design  is  to  be  extensible  and  open,  to  allow  COTS  packages  to  be  swapped  out  as  they  become 


4  “Gresham's  law”  is  named  for  Sir  Thomas  Gresham,  and  is  a  principle  of  economics  that  is  most 
commonly  stated  as,  “Bad  (overvalued)  money  drives  out  good  (undervalued)  money.”  It  refers  to  the  notion  that 
when  there  are  two  different  forms  of  currency  with  the  same  face  value,  the  currency  with  the  higher  intrinsic  value 
will  drive  the  currency  with  the  lower  intrinsic  value  out  of  circulation. 
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obsolete  and  need  to  be  replaced.”  The  program  was  looking  into  having  a  government  logistics 
depot  sustain  the  deployed  system  while  the  contractor  continued  to  develop  the  next-generation 
version  of  the  system. 

Hints  of  Trouble 

Planning  for  sustainment  of  the  system  was  only  one  of  many  responsibilities  on  the 
PMO’s  plate.  People  familiar  with  sustainment  knew  that  “A  good  sustainment  plan  will  consist 
of  several  different  plans,  including. . .  how  to  do  licensing  upgrades,  etc.”  However,  things  had 
not  been  going  smoothly  with  development,  and  there  was  a  great  deal  of  pressure  on  both  the 
PMO  and  the  development  team  to  deliver  the  new  version,  pushing  longer-term  concerns  like 
sustainment  to  the  back.  The  PMO  investigated  options  to  use  the  development  contractor  for 
sustainment,  go  entirely  with  a  government  depot  sustainment  team,  or  create  a  partnership  with 
some  combination  of  government  and  contractor. 

As  deployment  of  the  initial  system  finally  began  to  near,  and  discussions  began  on 
moving  its  sustainment  to  a  separate  government  team,  the  PMO  discovered  that  some  key 
decisions  hadn’t  even  been  made  internally,  much  less  discussed  with  the  prospective 
sustainment  team.  Despite  the  fact  that  the  use  of  COTS  products  was  a  key  part  of  the 
development  approach,  the  potential  lead  for  the  new  government  sustainment  team  at  the  depot 
realized  that,  “We’ve  never  considered  that  by  centrally  buying  licenses  for  COTS  products  we 
would  be  able  to  get  economies  of  scale,  and  save  government  money.  From  my  standpoint,  that 
would  just  make  things  harder  for  us,  even  though  it  would  save  money  for  the  taxpayers.” 
Additionally,  the  depot  sustainment  team  pointed  out  that  “We’re  still  waiting  for  an  updated 
version  of  the  [PMO  sustainment  planning]  document,  which  isn’t  completed,”  and  noted  that 
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“The  [sustainment  plan]  should  have  been  signed  when  they  went  into  LRIP5  production  two 
years  ago.” 

Unintended  Consequences 

The  consequences  of  a  lack  of  early  sustainment  planning  became  more  apparent  as  the 
program  neared  completion  of  the  latest  release  of  the  production  system.  Not  only  were  there 
issues  with  the  sustainment  plan  in  general,  but  there  were  fundamental  problems  on  the  program 
side  as  well.  The  contractor  program  manager  pointedly  observed  that  the  depot  sustainment 
team  “. .  .has  no  capability  to  sustain  [the  system],  because  there  is  little  in  the  way  of  a  technical 
data  package  to  hand  over”  to  them,  especially  in  the  form  of  system  documentation.  The 
prospective  sustainment  depot  realized  that  “As  far  as  other  things  we’d  need  to  have  delivered 
to  be  able  to  sustain  [the  system],  we  need  the  source  code,  the  development  tools  used,  and 
advice  from  [the  contractor]  on  their  experience  in  building  the  code.”  The  contractor  was  well 
aware  of  what  was  needed  to  do  sustainment  properly — but  felt  no  responsibility  to  supply  it  as 
the  government  had  not  required  it,  and  had  little  incentive  even  to  point  this  shortcoming  out  to 
the  government  while  there  was  still  time  to  address  it. 

One  of  the  most  serious  disconnects  occurred  when  it  was  realized  that  the  tools  the 
contractor  had  been  using  to  track  installations  and  versions  in  the  field — not  to  mention  the 
defect  database,  the  action  item  database,  the  engineering  drawing  database,  and  1 1  other 
databases — were  mostly  proprietary  to  the  contractor,  even  though  they  would  be  needed  for 
sustainment.  The  contractor  admitted  that  “The  database  is  owned  by  [us],  but  the  information  is 
owned  by  the  government.  We  hadn’t  thought  about  how  to  give  the  government  the  content  in 


5  Low  Rate  Initial  Production 
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the  STR6  database,  and  there  is  no  government  deliverable  to  turn  this  over.  We  could  dump  it  all 
into  hardcopy,  but  we  have  no  tasking  to  provide  it  in  a  nice,  accessible  format.”  In  fact,  “If  we 
were  asked  to  pack  all  of  this  up,  it  would  take  time.  It  took  months  to  do  a  conversion  via  scripts 
from  the  old  tool  to  move  to  the  STR  tool.  Further,  even  if  the  government  wanted  STR,  they 
probably  couldn’t  buy  it”  because  it’s  a  proprietary  in-house  tool  that  belongs  to  the  contractor. 

In  fact,  “There  may  be  15  databases  in  total,  and  you  would  definitely  need  all  of  these,  and  their 
files  in  their  office  file  cabinets,  to  hand  over  to  a  sustainment  organization  so  that  they  had 
proper  records.” 

Even  if  all  of  the  development  tools  had  been  free,  or  if  funding  to  purchase  them  were 
readily  available,  there  was  still  the  issue  of  the  system  expertise  of  the  sustainment  organization. 
While  very  capable  in  general,  most  agreed  that  the  depot  would  need  to  hire  some  of  the 
development  contractor  people  with  critical  knowledge  in  order  to  keep  them  available,  and  help 
the  depot  folks  learn  how  to  do  maintenance  on  the  system.  The  contractor  sustainment  people 
were  “. .  .bright,  dedicated. . .  no  problems  with  them.  They  know  all  the  parts  of  the  system. 
Anyone  else  coming  in  would  have  to  hire  those  guys  because  of  their  knowledge.”  However, 
the  contractor  was  “. .  .already  losing  people,  and  they’re  behind  the  power  curve  in  retaining 
them.”  While  there  was  training  available  for  users  of  the  system,  “...there’s  nothing  for 
maintainers  or  developers.”  The  government  “. .  .required  that  a  [contractor]  retention  plan  be  put 
in  place,  to  use  money  they  didn’t  earn  in  the  award  fee  to  retain  employees.  But. . .  it’s  not 
enough  money  to  keep  50  people  on — it’s  only  $1,000  per  person.” 


6  Software  Trouble  Report — a  software  defect  or  discrepancy  (bug)  report 
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One  of  the  most  important  things  for  a  sustainment  organization  to  have  when  taking  on  a 
system  is  a  comprehensive  set  of  internal  system  documentation — a  “technical  data  package.”  As 
one  PMO  staff  member  described  it,  “For  a  system  to  be  ready  to  transition  to  sustainment  I 
would  expect  to  see  well-documented  code,  all  software  development  folders  /  notebooks, 
complete  architecture  pictures,  etc.,  so  you  could  turn  it  all  over.  This  is  not  all  available  on  [this 
program],  because  it  wasn’t  paid  for.  I  do  think  that  [the  contractor]  has  some  of  these  things  that 
were  developed  on  their  own  nickel,  and  the  government  could  probably  contract  to  purchase 
them. . .”  The  salient  question  might  be,  at  what  price  could  it  be  negotiated? 

Contractor  Logistics  Support 

The  PMO  was  disheartened  to  see  the  lack  of  proper  preparation  that  had  been  made  to 
move  the  system  into  sustainment.  As  one  program  official  wryly  phrased  the  solicitation,  “The 
[PMO]  RFI  on  sustainment  said  that  there  are  no  specifications  for  the  system,  but  would  you 
perhaps  be  interested  in  sustaining  it?” 

The  PMO  knew  that  the  development  contractor  would  be  very  happy  to  negotiate  with 
the  government  to  offer  sustainment  services  for  the  system  as  well — and  given  their  unique 
qualifications,  and  the  challenges  facing  any  other  sustainment  organization  that  might  be 
capable  of  the  work,  they  were  in  a  very  strong  (and  almost  non-competitive)  bargaining 
position. 

Locked  In 

The  program  office  began  to  realize  that  the  situation  they  had  allowed  to  happen  had 
neatly  painted  them  into  a  comer.  Regarding  the  databases,  they  had  few  good  answers  to  the 
PMO’s  question,  “Where  will  the  data  reside  if  [the  contractor]  goes  away?”  The  PMO  admitted 
that  the  contractor  folks  “. .  .are  the  only  people  who  can  sustain  this  program,  because  they’re 
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the  only  ones  who  know  the  software  well  enough.”  While  at  first  it  appeared  to  be  a  viable 
solution,  it  ultimately  proved  to  be  infeasible  to  try  to  arrange  for  government  depot  sustainment 
when  little  of  the  proper  planning  had  been  done.  It  would  be  too  expensive,  and  too  time- 
consuming  to  transfer  sustainment  responsibility  to  a  new  organization  at  this  late  stage — thus 
losing  the  opportunity  to  compete  the  maintenance  effort. 

In  the  end,  there  was  little  surprise  to  anyone  in  the  program  office  when  they  found 
themselves  “. .  .working  on  the  J&A  (Justification  and  Approval7).”  When  they’d  looked  at  all  of 
the  issues  with  transferring  sustainment  responsibility,  “Because  the  risk  was  so  high,  they 
decided  to  go  back  to  [the  contractor]  as  sole  source”  for  sustainment. 

“Sacrificing  Sustainment”  Scenario  Analysis 

This  scenario  is  a  case  of  benign  neglect  on  the  part  of  an  overworked  PMO,  and  rational 
self-interest  on  the  part  of  a  profit-seeking  contractor.  The  PMO  redirects  some  of  its  staff  from 
longer-term  activities  such  as  sustainment  planning  to  focus  on  near-term  needs  instead.  When 
the  program  finally  nears  completion,  the  government  realizes  that  the  lack  of  sufficient 
sustainment  planning  up-front  means  that  they  have  little  ability  to  competitively  and  fairly  bid  a 
contract  for  system  sustainment.  They  have  no  choice  but  to  negotiate  additional  sustainment 
services  with  the  development  contractor,  and  so  the  contractor  is  now  effectively  “locked  in”  for 
sustainment. 

Figure  2  depicts  the  basic  causal  interactions  and  flows  of  the  “Sacrificing  Sustainment” 
dynamic. 


7  Justification  and  Approval  is  used  to  give  the  justification  for  a  proposed  sole  source  acquisition  award. 
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Figure  2:  Overview  of  the  “Sacrificing  Sustainment”  Dynamic 8 

The  initial  Cost  and  Schedule  Pressure  that  the  program  is  facing  drives  a  decision:  either 
to  choose  to  invest  time  and  effort  to  Implement  Cost-Effective  Sustainment  (B2),  or  to  Divert 
Effort  from  Sustainment  Planning  (Bl).  Either  option  will,  ultimately,  help  to  reduce  Cost  and 
Schedule  Pressure ,  but  in  different  timeframes,  and  with  different  longer-term  consequences. 

The  PMO  needs  to  free  up  staff  to  address  other  near-term  priorities,  and  chooses  to  Divert  Effort 
from  Sustainment  Planning  (Bl).  However,  after  a  delay,  this  means  that  there  have  been  fewer 
Actions  Completed  to  Compete  Sustainment ,  which  reduces  the  number  of  Viable  Sustainment 

8  The  “Sacrificing  Sustainment”  dynamic  is  an  adaptation  of  the  “Shifting  the  Burden”  systems  archetype 
as  published  in  Kim  (Kim,  1993). 
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Options  (R).  This  in  turn  reduces  the  PMO’s  Capacity  for  Competing  Sustainment,  which 
increases  reliance  on  Sole  Source  Sustainment  with  Development  Contractor,  drives  up 
Sustainment  Cost  due  to  their  poor  negotiating  position,  and  then  lowers  the  ability  to  Implement 
Cost-Effective  Sustainment,  as  the  program  is  left  in  a  sole-source  situation  with  little  leverage 
over  the  contractor.  Without  a  more  cost-effective  sustainment  alternative,  after  a  delay  the  Cost 
and  Schedule  Pressure  increases  due  to  higher-than-expected  sustainment  costs. 

“Sacrificing  Sustainment”  Scenario  Implications  for  Competition 

The  outcome  for  the  sustainment  approach  was  the  result  of  a  litany  of  many  different 
issues,  with  no  one  problem  being  overwhelming  in  and  of  itself,  but  collectively  producing  an 
inevitable  result.  One  concern  is  that  there  is  a  clear  misaligned  incentive  in  encouraging  an 
overworked  PMO  staff  to  focus  on  near-term  issues  at  the  expense  of  longer-term  strategic  ones. 
This  manifests  itself  at  an  individual  level  when  one  of  the  PMO  sustainment  planners  points  out 
that  they  are  making  a  conscious  trade-off  between  what  would  be  best  for  them  personally  (less 
effort  by  not  centralizing  COTS  license  purchases),  vs.  what  would  be  best  in  the  longer  term  for 
the  program  and  the  taxpayers  (better  negotiating  position  to  reduce  COTS  license  costs).  As  the 
causal  loop  diagram  shows,  and  DoD  guidance  recommends,  making  sustainment  plans  and 
decisions  needs  to  be  done  continually  from  the  start  of  the  effort.  If  such  decisions  are  deferred, 
they  make  the  sustainment  options  both  fewer  and  more  risky  for  any  choice  other  than  using  the 
original  system  contractor. 

What  is  clear  is  that  when  software  development  files,  test  drivers  and  results,  COTS  and 
proprietary  software  development  tools,  and  other  artifacts  are  not  provided  to  the  PMO,  it  will 
be  costly  either  to  bring  on  a  different  contractor  for  sustainment,  or  to  do  it  organically. 
Precisely  why  this  situation  originally  occurs,  however,  is  less  clear.  Is  it  because  the  PMO 
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doesn’t  have  the  foresight  to  demand  it,  or  because  the  PMO  staff  hasn’t  fully  thought  through 
sustainment  when  the  contract  hasn’t  been  awarded  yet?  Is  it  also  partially  the  result  of  a 
“perverse  incentive”  on  the  part  of  the  development  contractor  not  to  help  the  PMO  make 
adequate  provisions  for  cost-effective  organic  sustainment,  such  as  providing  full  government 
data  rights,  and  not  using  proprietary  sustainment  tools — in  order  to  increase  the  likelihood  of 
being  awarded  a  contract  for  sustainment?  Regardless  of  the  reason,  when  the  system  is  fielded, 
the  program  office  will  likely  have  to  continue  to  work  with  the  development  contractor  because 
they  are  the  only  ones  with  the  expertise  necessary  to  build  and  install  upgrades  to  the  system. 
They  have  inadvertently  created  a  switching  barrier — by  neglecting  to  require  these  things,  the 
government  has  inadvertently  locked  the  development  contractor  into  doing  the  sustainment. 

To  clarify  the  relationships  among  all  of  these  ideas,  it  may  be  useful  to  link  them 
together.  In  the  “Sacrificing  Sustainment”  scenario,  there  are  incentives  for  PMO  management  to 
meet  the  immediate  PMO  goals  and  deliverables  with  respect  to  such  aspects  as  finances, 
contracts,  and  schedules  (i.e.,  recognition  and  promotion),  as  well  as  disincentives  to  not  meeting 
them  (i.e.,  damage  to  reputation).  However,  the  reality  is  that  (as  is  frequently  the  case)  there  is 
insufficient  PMO  staff  to  meet  those  deadlines.  In  addition,  there  are  regular  18-24  month 
rotations  of  key  PMO  managers  which  limit  their  tenure  on  the  program.  Given  the  conflict 
between  insufficient  staffing  and  looming  deadlines,  PMO  management  could  choose  to 
acknowledge  an  impending  schedule  slip,  or  perhaps  request  an  increase  in  the  size  of  the  PMO 
staff.  However,  given  the  implicit  disincentives  to  pursuing  either  of  these  options  (i.e.,  damage 
to  reputation,  perception  of  incompetence),  the  actual  decision  by  PMO  management  is  to  divert 
PMO  staff  from  their  longer-term  sustainment  planning  to  work  on  the  immediate  goals  and 
deliverables.  This  is  a  classic  “social  trap”  of  trading  off  long-term  interests  for  short-term 
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priorities.  The  short-term  result  is  that  the  immediate  goals  and  deliverables  are  met — but  the 
longer-term  results  are  that  sustainment  planning  is  neglected,  and  as  a  result  the  development 
contractor  is  locked  in  for  sustainment. 

There  are  various  places  during  the  lifecycle  of  government  systems  that  competition  can 
be  used  to  improve  quality  and  reduce  costs.  The  use  of  systems  thinking  can  help  to  show 
possible  reasons  why  some  opportunities  for  competition  may  be  lost.  If  PMO  management  can 
better  understand  the  impact  of  their  early  decisions  during  both  contracting  and  execution,  it 
might  be  possible  to  better  leverage  existing  opportunities  to  introduce  competition  into 
sustainment. 

Conclusions  Regarding  Competition  in  Acquisition 

There  are  some  significant  conclusions  that  can  be  drawn  from  an  examination  of 
misaligned  incentives  and  competition  in  acquisition. 

First,  misaligned  incentives  often  occur  in  the  presence  of  conflicting  governance,  where 
governance  is  the  set  of  rules  that  control  an  activity,  along  with  the  rewards  or  punishments  for 
the  participants.  Unless  laws,  policies,  or  other  regulations  encourage  them  to  do  otherwise, 
people  tend  to  behave  in  their  own  self-interest.  That  said,  the  existence  of  an  (often 
unintentional)  incentive  embedded  within  the  acquisition  governance  to  follow  a  particular 
course  of  action,  especially  one  that  is  contrary  to  an  individual’s  best  interests,  does  not 
necessarily  mean  that  the  incentive  will  result  in  that  behavior.  However,  such  an  incentive  does 
create  an  ethical  dilemma  that  may  pressure  people  to  take  an  action  contrary  to  their  personal 
interests  in  order  to  do  what  is  best  for  their  program,  their  Service,  or  their  country.  Although 
many  people  believe  they  have  the  integrity  to  “do  the  right  thing”  in  such  a  situation,  the  point 
is  that  acquisition  program  management  and  staff  should  not  be  put  in  a  position  where  they  must 
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make  such  a  choice,  as  it  ultimately  subverts  both  personal  integrity  and  the  organization’s  goals. 
The  government  acquisition  community  needs  to  take  steps  to  adjust  governance  to  better  align 
the  personal  and  program-level  goals  of  acquisition  staff  with  the  longer-term  objectives  of 
government  acquisition.  Without  this  alignment  the  government  acquisition  system  will  continue 
to  rely  largely  on  the  integrity  of  the  participants — while  at  the  same  time  undermining  that  very 
integrity. 

A  second  point  to  be  made  is  that  market  competition  is  a  long-standing  and  effective 
mechanism  for  achieving  efficient  outcomes  for  economic  transactions.  As  a  result  it  is  the 
generally  recommended  approach  for  reducing  cost  and  improving  quality.  However, 
competition  isn't  always  as  effective  as  we  might  like  in  acquisition  contexts.  Markets  suffer 
from  what  economists  call  “externalities,”  where  there  are  side-effects  of  the  transaction  that 
impact  others  outside  the  immediate  buyer-seller  transaction  (e.g.,  air  and  water  pollution).  When 
economists  point  to  markets  as  being  “incentive-compatible  mechanisms”  (i.e.,  where 
participants  have  incentives  to  speak  and  behave  honestly),  the  perceived  efficiencies  of  markets 
do  not  account  for  externalities  because  those  costs  are  borne  by  others.  Again,  the  unintended 
consequences  of  our  decisions  and  actions  often  circle  back  through  feedback  to  affect  us  in 
unplanned  and  unexpected  ways — making  the  study  of  such  “side-effects”  vitally  important. 

Looking  at  the  bigger  picture,  in  scenario  #1  on  “Consolidation  of  Joint  Programs”  the 
forces  of  competition  contribute  to  causing  a  problem  by  driving  contractors  toward 
underbidding  with  its  resultant  impacts  on  program  performance.  In  scenario  #2  on  “Sacrificing 
Sustainment”  the  problem  is  the  lack  of  competition  that  could  have  helped  to  reduce  longer- 
term  sustainment  costs.  Competition  is  one  of  the  forces  driving  Adam  Smith’s  incentive- 
compatible  “invisible  hand”  of  rational  self-interest  in  markets  that  moves  actors  to  perform  their 
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tasks  to  our  collective  benefit,  without  requiring  a  sense  of  altruism  on  their  parts.  Competition 
is  neither  inherently  good  nor  bad,  but  it  must  be  governed  and  controlled  to  produce  only  the 
effects  we  desire.  This  is  the  beauty  of  aligned  incentives  -  when  they  have  been  properly 
employed,  the  behavior  we  desire  will  result.  The  difficulty  is  in  understanding  the  likely 
consequences,  both  intended  and  unintended,  of  those  incentives  when  they  are  part  of  a  highly 
complex  and  dynamic  system  with  autonomous  actors  engaged  in  multiple  levels  of  feedback 
from  many  different  sources.  The  qualitative  models  presented  here  are  useful  and  instructional 
within  their  limited  scope,  but  they  pale  in  comparison  to  the  complexity  present  in  actual 
acquisition  programs.  In  fact,  the  level  of  complexity  and  uncertainty  in  large  acquisition 
programs  makes  it  extremely  difficult  to  identify  optimal  solutions  analytically  through  such 
formal  approaches  as  game  theory  and  mechanism  design.  At  the  other  extreme,  it  is  neither 
feasible  nor  affordable  to  conduct  empirical  experiments  on  large-scale  acquisition  programs  in 
order  to  fully  understand  the  effects  of  incentives  on  behaviors  and  program  outcomes.  How, 
then,  are  we  to  make  effective  progress  in  dealing  with  these  challenges? 

Addressing  Misaligned  Incentives  in  Acquisition 

In  order  to  make  the  analysis  of  organizational  problems  based  on  misaligned  incentives 
more  tractable,  research  is  being  conducted  at  the  SEI  to  study  the  recurring  behaviors  of 
software -reliant  acquisition  programs  using  system  dynamics  modeling.  The  following  sections 
describe  the  direction  and  status  of  this  work. 

System  Dynamics 

As  valuable  as  the  systems  thinking  approach  can  be  in  describing  some  dynamics,  its 
usefulness  declines  quickly  as  the  complexity  of  the  dynamic  in  question  grows.  Systems 
thinking  is  beneficial  in  developing  a  shared  understanding  of  a  particular  behavior — but  this 
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value  is  lost  when  those  involved  are  no  longer  able  to  fully  comprehend  the  multiple  interacting 
loops  of  the  causal  loop  diagram.  As  shown  even  in  the  comparatively  small  scenarios  presented 
here,  systems  thinking  becomes  problematic  when  the  complexity  of  a  system  makes  it 
impossible  to  envision  how  the  dynamic  will  behave.  It  is  at  this  point  that  an  approach  such  as 
system  dynamics  may  need  to  be  employed. 

System  dynamics  is  a  method  for  modeling  and  analyzing  the  holistic  nature  of  complex 
problems  as  they  evolve  over  time  (Sterman,  2000).  Like  systems  thinking,  system  dynamics 
focuses  on  capturing  the  underlying  feedback  structure  of  a  system’s  behavior  in  an  effort  to 
understand  which  loop’s  influence  will  dominate  others  over  time.  System  dynamics  has  been 
used  to  gain  insight  into  some  of  the  most  challenging  strategy  questions  facing  both  government 
and  business  for  several  decades  now  (Sterman,  2000).  The  System  Dynamics  Society’s 
definition  of  system  dynamics  states  that  it  is  “...a  computer-aided  approach  to  policy  analysis 
and  design.  It  applies  to  dynamic  problems  arising  in  complex  social,  managerial,  economic,  or 
ecological  systems  -  literally  any  dynamic  systems  characterized  by  interdependence,  mutual 
interaction,  information  feedback,  and  circular  causality.”9  As  a  result,  system  dynamics  is 
particularly  useful  for  gaining  insight  into  difficult  management  situations  in  which  well- 
intentioned  efforts  to  solve  a  problem  can  potentially  make  it  worse. 

In  creating  a  system  dynamics  model,  all  of  the  elements  of  the  system  including  such  so- 
called  “soft”  aspects  as  policy,  procedural,  administrative,  cultural,  and  psychological  factors 
along  with  the  more  traditional  technical  factors  can  be  represented  and  processed.  While  the 
results  of  such  modeling  produce  imprecise  predictions  of  future  system  behavior.  Meadows 

9  This  definition  may  be  found  on  the  website  for  the  System  Dynamics  Society  at 
http://www.systemdynamics.org/whatjs_system__dynamics.htm 
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(Meadows,  1974)  points  out  that  “this  level  of  knowledge  is  less  satisfactory  than  a  perfect, 
precise  prediction  would  be,  but  it  is  still  a  significant  advance  over  the  level  of  understanding 
permitted  by  current  mental  models.”  This  is  precisely  the  point.  Despite  the  inevitable  flaws  and 
incompleteness  in  any  system  dynamics  model  compared  to  the  reality  it  attempts  to  represent, 
the  mental  models  that  are  presently  the  only  alternative  are  substantially  worse. 

Modeling  Acquisition  Program  Behaviors 

The  attributes  of  system  dynamics  modeling  described  above  make  it  a  natural  tool  of 
choice  to  model  the  behavior  of  acquisition  programs.  We  are  currently  working  to  model  the 
key  dynamic  behaviors  of  joint  programs  in  an  effort  both  to  1)  better  understand  the  nature  of 
the  dynamics  at  work  that  create  recurring  challenges  for  such  programs,  and  2)  identify 
potentially  effective  mitigation  strategies  that  can  be  applied  to  improve  program  outcomes.  In 
order  to  create  a  system  dynamics  model  that  is  as  accurate  as  possible,  groups  of  experienced 
acquisition  subject  matter  experts  are  being  used  to  conduct  group  modeling  sessions  to  elicit  the 
critical  elements  of  the  model,  and  define  the  nature  of  the  relationships  that  exist  among  the  key 
variables.  The  model  is  defined  using  a  commercial  system  dynamics  modeling  product  that  is 
able  to  simulate  its  behavior,  allowing  it  to  be  validated  against  both  qualitative  and  quantitative 
historical  data  collected  from  past  joint  programs.10 

The  model  explicitly  addresses  both  the  structural  dynamics  and  the  misaligned 
incentives  aspects  of  the  acquisition  program  being  modeled.  The  activity  of  the  contractor’s 
software  development  team  represents  a  set  of  underlying  structural  dynamics  that  both 
influence,  and  are  influenced  by,  the  incentives  that  drive  the  decision-making  performed  by  the 

10  This  work  is  based  on  a  prior  effort  using  system  dynamics  modeling  to  characterize  another  recurring 
acquisition  dynamic  titled  “The  Evolution  of  a  Science  Project,”  as  described  in  Novak  (Novak,  Moore  &  Alberts, 
2012). 
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key  stakeholders  of  the  program.  As  we  have  seen  in  the  scenarios  presented,  key  concepts  such 
as  fairness,  cooperativeness,  and  self-interest  all  come  into  play  in  these  decisions,  and  can  be 
primary  drivers  of  the  outcomes.  Once  the  decision-making  models  are  validated,  candidate 
solution  models  that  apply  behaviors  to  mitigate  the  problematic  dynamics  are  integrated  with 
the  model  to  evaluate  their  effectiveness  in  reducing  the  behavior.  When  these  trials  are 
complete,  if  a  candidate  solution  strategy  has  been  validated  and  shows  promise,  it  may  merit  a 
pilot  implementation  to  see  if  its  effectiveness  can  be  reproduced  on  an  actual  acquisition 
program. 

It  should  be  remembered  that  any  model  of  an  aspect  of  the  real  world  will  be  incorrect 
because  it  cannot  represent  all  of  the  detail  involved.  This  raises  legitimate  questions  about  the 
validity  and  usefulness  of  the  model.  It  is  certainly  essential  to  formally  validate  any  model  that 
will  be  used  to  model  behaviors  and  predict  outcomes,  but  even  with  these  precautions  the  model 
will  necessarily  be  incomplete.  Although  all  models  are  inherently  flawed,  we  still  believe  they 
provide  important  insights,  point  out  discrepancies,  and  raise  valuable  questions — and  are 
significantly  more  useful  in  understanding  complex  system  behaviors  than  having  no  model  at 
all. 

Impacts  on  Acquisition  Decision-Making 

Once  the  underlying  reasons  for  counter-productive  dynamic  behaviors  in  acquisition 
programs  have  been  uncovered,  acquisition  leaders  and  staff  should  be  educated  so  that  they  are 
capable  of  recognizing  when  such  situations  are  present  in  actual  programs,  and  are  aware  of  the 
solutions  that  can  be  applied  to  mitigate  them.  As  the  Task  Force  on  Defense  Acquisition  Law 
and  Oversight  ("Getting  to  best:,"  2009)  acknowledged,  “...the  government  too  often  finds  itself 
with  minimally  experienced  and  transient  individuals  leading  major  acquisition  programs...” 
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Because  of  this  it  is  important  to  try  to  significantly  improve  the  decision-making  abilities  of 
acquisition  program  office  staff  in  these  areas.  In  another  aspect  of  the  research,  this  work  will 
use  system  dynamics  models  to  create  interactive  classroom  games  of  recurring  acquisition 
behaviors  that  can  provide  acquisition  staff  with  an  experiential  learning  environment.  This 
approach  will  allow  them  to  explore  these  dynamics  and  leam  about  their  properties  for 
themselves,  which  maximizes  both  learning  and  retention  (Oh  Navarro,  2006;  Pfahl, 
Laitenberger,  Ruhe,  Dorsch  &  Krivobokova,  2004;  Pfahl,  Ruhe,  Lebansft  &  Stupperich,  2006), 
and  thus  helps  to  compensate  for  the  present  variance  in  program  management  office  staff 
experience  across  programs. 

Implications  for  Acquisition  Policy 

Using  an  actual  acquisition  program  (or  the  entire  acquisition  system)  to  test  the  effects 
of  new  policies  is  not  desirable,  but  has  been  employed  due  to  the  lack  of  good  alternatives. 
Fortunately,  system  dynamics  modeling  provides  a  feasible,  cost-effective  way  to  conduct 
experiments  regarding  the  efficacy  of  proposed  policies,  and  can  be  used  to  help  lawmakers, 
policy  makers,  and  acquisition  professionals  come  to  informed  conclusions  regarding  the  relative 
consequences  of  different  potential  decisions.  Modeling  the  acquisition  context  allows  us  to 
create  a  “testbed”  that  can  be  used  to  help  forecast  possible  specific  and  broader  effects  of 
decisions  and  actions,  and  explore  ways  to  improve  the  overall  acquisition  process.  In  addition,  it 
can  give  inexperienced  acquisition  leaders  and  staff  the  chance  to  experiment  in  a  safe 
environment  where  a)  the  near  and  longer-term  consequences  of  actions  can  be  seen  both  quickly 
and  more  clearly,  and  b)  some  of  the  many  possible  side-effects  of  their  actions  can  be 
understood,  and  thus  potentially  avoided  in  actual  practice. 
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Summary 

Government  acquisition  is  difficult.  A  single  large  buyer  commanding  vast  sums  of 
money  with  which  to  address  some  of  the  most  complex  problems  known  to  us  is  a  unique 
situation  in  our  society.  Understanding  how  to  improve  the  process  that  this  single  buyer  uses  to 
purchase  state-of-the-art  systems  is  a  daunting  challenge.  We  believe  that  modeling  the 
incentives  and  dynamic  behaviors  of  acquisition  programs  is  one  of  the  most  powerful  and  cost- 
effective  ways  available  to  the  defense  acquisition  community  of  understanding  problems, 
evaluating  candidate  solutions,  and  producing  better  acquisition  program  outcomes.  Even  minor 
efficiencies  gained  in  large  acquisition  programs  from  using  these  methods  could  produce 
benefits  in  functionality,  quality,  performance,  improved  time-to-deployment,  and  direct  cost 
savings  that  would  recoup  the  investment  many  times  over.  By  using  systems  thinking  and 
system  dynamics  models,  new  ways  of  using  competition  can  be  explored  in  the  lab  and  virtual 
worlds  before  trying  them  out  on  ourselves. 
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Acquisition  Programs  are  Dynamic  Systems 

Complex  Interactions:  Interactions  between  acquisition  stakeholders  are  non¬ 
linear  because  of  the  presence  of  feedback 

•  What  you  do  depends  on  what\_  do,  which  depends  on  what  you  do... 

Non-linear  Behavior:  Non-linear  behavior  defies  traditional  mathematical  analysis 


Sensitivity  to  Initial  Conditions:  Results  may  vary  greatly  due  to  seemingly 
insignificant  differences  in  the  starting  point(s) 

Organizational:  Key  issues  in  software  acquisition  are  often  management  and 
organization-related  —  not  technical 

•  “No  matter  what  the  problem  is,  it’s  always  a  people  problem.” 

— Gerald  Weinberg 

Partitioning:  Partitioning  isn’t  possible  when  there  are  complex  interactions 
between  components 
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Problem 


Poor  acquisition  program  performance  inhibits  military  performance  by 
depriving  the  warfighter  of  critical  systems  to  achieve  mission  objectives 

•  Delayed  systems  withhold  needed  capabilities 

•  Wasted  resources  drain  funding  needed  for  new  systems 

Acquisitions  fail  for  both  technical  and  non-technical  reasons;  people  issues 
often  drive  adverse  acquisition  dynamics 

•  Human,  organizational,  and  management  issues  drive  poor  program  performance 

Acquisition  programs  are  complex  systems  with  structural  dynamics 

•  Feedback  in  acquisition  produces  non-linear  interactions  (feedback)  that  add  complexity 

•  Complex  systems  can  produce  seemingly  unpredictable  behaviors 

Misaligned  incentives  are  a  key  driver  of  poor  acquisition  outcomes 

•  “Social  dilemmas”  are  a  major  category  of  misaligned  incentives  that  have  received  much  study 

•  Social  dilemmas  occur  frequently  in  software-reliant  acquisition  programs 
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Structural  Dynamics  in  Acquisition 

Structural  dynamics  are  the  natural  and  “physical”  processes  involved  in 
carrying  out  a  system’s  function 

Unrecognized  structural  feedback  dynamics  underlie  acquisition,  and  can 
drive  complexity  and  adverse  acquisition  behaviors 

Example:  Long  Program  Duration  Grows  Schedule  (Longer  Begets  Bigger) 

•  Long  duration  allows  greater  capability  to  be  built 

•  Long  duration  drives  use  of  immature  technology  to  avoid  obsolescence 

•  Long  duration  drives  scope  creep  due  to  changing  threats  and  new  technologies 


Complex  feedback  and  delays  make  the  management  and  control  of  acquisition  programs  difficult 
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Misaligned  Incentives  in  Acquisition 

Structural  reasons  like  feedback  and  delays  aren’t  the  only  causes  for 
acquisition  failure — incentives  play  a  key  role  as  well. 

Misaligned  incentives  occur  when: 

•  Lower-level  individual  goals  conflict  with  group  goals 

•  Short-term  goals  conflict  with  long-term  goals 

The  result  is  that: 

•  Some  group  goals  only  succeed  at  the  expense  of  individual  goals 

•  Some  long-term  goals  only  succeed  at  the  expense  of  short-term  goals 

Some  acquisition  programs  are  prevented  from  succeeding  for  structural 
and  incentive  reasons — not  poor  work  or  lack  of  effort. 


Key  Idea 


Misaligned  incentives  can  push  people  to  make  impossible  choices  and  trade-offs 
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Misaligned  Incentives  in  Acquisition 

Misaligned  incentives  in  acquisition  programs  put  individual  or  program- 
specific  interests  ahead  of  PEO  or  Service  interests,  turning  cooperation 
into  opposition 

Examp/e:  Joint  Programs 

•  To  meet  conflicting  requirements,  cost,  schedule,  size,  complexity,  risk  all  go  up 

•  Users  prefer  custom  solutions  they  control  that  are  certain  to  meet  their  needs 

Examp/e:  Shared  Infrastructure  Development 

•  Programs  have  an  incentive  to  wait  for  another  program  to  use  the  shared 
infrastructure  first — better  that  others  work  out  the  problems,  than  risk  failure  of 
the  program 


Key  Idea 


Misaligned  incentives  are  pervasive  in  contributing  to  poor  outcomes  for  acquisition  programs 


□ 


SEI  Proprietary;  Distribution:  Director’s  Office  Permission  Required 


=  Software  Engineering  Institute  CarnegieMellon 


>2012  Carnegie  Mellon  University 


The  Effects  of  Incentives  in  Acquisition  Competition  on  Program  Outcomes 

Misaligned  Incentives  in  Acquisition 

Risk:  Low  incentive  to  identify  program  risks  if  it  can  adversely  affect  personal  standing. 

Defects:  Incentives  to  find  defects  can  result  in  the  intentional  insertion  of  defects. 

Schedule:  Incentives  to  improve  performance  by  meeting  a  set  date  can  mean  quality 
processes  are  sacrificed  to  meet  that  date. 

Technology:  Incentives  to  use  risky,  immature  technology  to  achieve  better  system  capability, 
and  give  good  experience  to  the  contractor. 

Contracts:  Incentives  to  drag  out  development  on  CP  &  T&M  contracts  to  increase  profits. 
Staffing:  Incentives  to  slow  efforts/stretch  schedule  if  there’s  no  next  project  to  move  on  to. 
Cancellation:  Low  incentive  to  cancel  ailing  programs  if  it’s  not  in  interests  of  staff. 

Scope:  Low  incentive  for  users  to  ask  for  only  minimal  system  capability  if  it’s  free  to  them. 


Key  Idea 


Misaligned  incentives  occur  every  day  in  acquisition  programs 
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Challenge 

Acquisition  leaders  may  have  inadequate  decision-making  experience 

•  May  be  insufficiently  trained  in  making  key  acquisition  decisions 

•  Lacking  software  acquisition  experience,  they  are  unaware  of: 

•  The  complexity  of  acquisition  programs 

•  The  unintended  consequences  of  many  decisions  made  on  programs 

Education  is  the  best  alternative — but  conventional  training  is  ineffective  for 
decision-makers  in  dynamically  complex  domains 

•  Traditional  education  methods  may  not  translate  well  to  acquisition  realities 

•  Well-intentioned  decisions  are  undermined  by  complexity  and  adverse  unintended 
consequences 

•  Poor  acquisition  management  has  major  cost,  schedule,  and  quality  impacts 

•  Improved  decision-making  requires  different  mental  models  [Shute  2009] 


Software  Engineering  Institute 
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Systems  Thinking 

Systems  Thinking  is  a  method  for  analyzing  complex  systems 

Developed  by  Jay  W.  Forrester  at  MIT  while  modeling  electrical  feedback 

•  Also  recognized  in  economic,  political,  business,  and  organizational  behaviors 

Uses  feedback  loops  to  describe  and  analyze  common  system  structures 
that  spin  out  of  control,  or  regulate  themselves 

Relationships  between  reinforcing  feedback  loops  and  balancing 
feedback  loops  drive  the  behavior  of  the  system 

Time  delays  obscure  the  connections  in  cause-and-effect  relationships 

•  Time  delays  in  feedback  affect  the  way  the  system  behaves 

•  People  are  poor  at  controlling  systems  with  long  time  delays 


Software  Engineering  Institute 
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Causal  Loop  Diagrams  (CLDs) 

Depict  qualitative  “ influencing’  relationships  (increasing  or  decreasing) 
and  time  delays  between  key  variables  that  describe  the  system 

Show  relationship  direction  by  labeling  them  Same  (+)  or  Opposite  (-) 
to  indicate  how  one  variable  behaves  based  on  the  previous  variable 

Consist  primarily  of  two  types  of  feedback  loops: 


•  Reinforcing  -  Changes  to  variables  reinforce,  moving  in  one  direction 

•  Balancing  -  Changes  to  variables  alternate ,  achieving  equilibrium 


Software  Engineering  Institute 
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“Sacrificing  Sustainment” 


Divert  Effort" 
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Actions 
Completed  to 
Compete 
Sustainment 
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Sustainment 

Cost 


Viable 

Sustainment 

Options 


Capacity  for 
Competing 
Sustainment 


Sole  Source 
Sustainment 
with 


Development 

Contractor 


S 


Diverting  Effort  from  Sustainment 
Planning  reduces  Viable 
Sustainment  Options,  along  with 
a  program’s  Capacity  for 
Competing  Sustainment.  This 
makes  Sole  Source  Sustainment 
with  the  Development  Contractor 
more  likely,  driving  up 
Sustainment  Cost,  and,  after  a 
delay,  increasing  Cost  and 
Schedule  Pressure. 
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Joint  Programs 


1.  A  JPO  PM  has  six 
stakeholder  programs 
planning  to  use  their 
joint  infrastructure 
software... 


7.  As  cost  escalates 
and  schedules  lengthen, 
participation  in  the 
joint  program  unravels 
and  collapses. 


6.  With  one  stakeholder 
gone,  the  amortized  costs 
for  the  other  programs 
increase  further — and 
another  program  leaves. 


2.  ...but  each  program 
demands  at  least  one 
major  feature  be  added 
to  the  software  just 
for  them. 


5.  As  the  schedule 
slips,  one  program 
decides  to  leave  the 
joint  program  and 
develop  its  own 
custom  software. 


4.  The  additional  design 

3.  The  JPO  agrees  to  the 

This  scenario  aqqreqates  v  \ 

changes  and  coding 

additional  requirements,  for 

three  SEI  software-reliant 

significantly  increase 

fear  of  losing  stakeholders 

system  acquisition  ITAs 

total  cost,  schedule, 

(who  could  build  custom  software). 

conducted  in  2006-2009. 

complexity,  and  risk. 
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“Consolidation  into  Joint  Programs” 
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“Consolidation  into  Joint  Programs”  2 
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The  consolidation  of  multiple 
needs  into  single  joint  acquisition 
programs  that  contractors  must 
then  underbid  to  win  creates 
schedule  pressure  that  drives 
cost  and  schedule  overruns,  and 
encourages  stakeholder 
programs  to  opt  out,  undermining 
the  purpose  and  value  of  the  joint 
program,  and  potentially  causing 
it  to  fail. 
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Misaligned  Incentives  in 
Acquisition 
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Research  Approach 


General 

Qualitative  Model 


Unintended 

Consequences 


Independent  Technical 

Assessment  (ITA)  Data 

Detailed  examinations  of 
challenged  programs 
with  interviews, 
document  reviews,  and 
code  analysis 
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Acquisition 
Problem  Model 


Deep 

Understanding  of 
Dynamic 
Acquisition 
Behavior 


Model-Based 
Simulation  of 
Potential 
Solutions 


Acquisition 
Qualitative  Model 


Foundation  for 
Acquisition 
Instructional 
Simulations 


Firefighting:  If  design  problems  are 
found  in  the  current  release,  more 
resources  must  be  used  to  fix  them. 
This  reduces  problems,  but  now  less 
work  is  done  on  the  next  release. 
This  undermines  its  early 
development  work,  and  increases 
design  problems  in  the  next  release. 
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Research  Approach  2 

Interactive 
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Key  Idea 


Interactive 

Participation 


Experiential  learning  can  significantly  improve  learners’  mental  models  and  their  acquisition  decision-making  | 
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Strategy 
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Key  Incentives  Behind  Acquisition  Failure 

Immature  Technology 

•  Government  prefers  providing  greatest  capability,  which  requires  latest  technologies 

•  Contractors  prefer  using  latest  technologies  to  boost  staff  competency  for  future  bids 

Joint  Programs 

•  To  meet  conflicting  requirements,  cost,  schedule,  size,  complexity,  and  risk  all  go  up 

•  Users  prefer  custom  solutions  they  control  that  are  certain  to  meet  their  needs 

Long  Duration 

•  Long  duration  allows  greater  capability  to  be  built 

•  Long  duration  drives  use  of  immature  technology  to  avoid  obsolescence 

•  Long  duration  drives  scope  creep  due  to  changing  threats  and  new  technologies 

•  Contractors  prefer  the  stability  and  revenue  of  longer  programs 

Turnover  and  Inexperience  in  Acquisition  Program  and  Technical  Management 

•  Personnel  on  short  rotations  may  not  be  invested  in  decisions  about  long-term  needs 

•  More  difficult  for  government  to  hire  and  retain  highly  experienced  personnel 

Unrealistic  Estimates  and  Underbidding 

•  Government  wants  low  cost  estimates  to  get  programs  approved 

•  Contractors  want  low  bids  to  win  contracts 
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“Firefighting”  Interactive  Exercise 
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Related  Disciplines  and  Concepts 


The  Acquisition  Dynamics  work 
starts  with  real  acquisition 
problems,  and  then  draws  on 
ideas  and  concepts  from  a 
variety  of  different  disciplines: 

Social  Science 

Game  Theory 

Social  Psychology 

Political  Science 

Economics 
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Summary 

Many  recurring  patterns  of  adverse  acquisition  behavior  can  be  explained  by 
structural  dynamics  and  misaligned  incentives 

Sacrificing  longer-term  planning  in  favor  of  nearer-term  priorities  inadvertently 
undermines  ability  to  compete  sustainment,  with  likely  longer-term  adverse 
effects  of  increasing  costs 

Consolidating  multiple  needs  into  single  joint  acquisition  programs  promotes 
underbidding,  inadvertently  fostering  cost  and  schedule  overruns  that  undermine 
the  joint  effort 

Use  executable  acquisition  models  to  analyze  known  adverse  software 
acquisition  dynamics,  and  test  proposed  mitigations/solutions 

•  Turn  existing  software  acquisition  domain  expertise  into  a  more  usable  form 

•  Apply  both  new  and  known  solutions  to  solving  recurring  dilemmas  in  acquisition 

Provide  experiential  learning  to  DoD  acquisition  staff  through  hands-on 
simulations  of  key  recurring  acquisition  dynamics 

•  Understand  common  side-effects  of  decisions  that  lead  to  poor  performance 

•  Let  acquisition  staff  gain  experience  through  education—  not  costly  mistakes 
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For  Additional  Information 


SEI  Report:  “ The  Evolution  of  a  Science  Project:  A  Preliminary  System  Dynamics 
Model  of  a  Recurring  Software-Reliant  Acquisition  BehavioP 

SEI  Report:  “ Success  in  Acquisition:  Using  Archetypes  to  Beat  the  Odds” 


SEI  Blog:  “Themes  Across  Acquisition  Programs”:  Parts  1-4 

Website:  http://www.sei.cmu.edu/acquisition/research/archetypes.cfm 

Download  all  twelve: 

PMO  vs.  Contractor  Hostility 
Underbidding  the  Contract 
Everything  for  Everybody 
The  Bow  Wave  Effect 
Brooks'  Law 
Firefighting 
"Happy  Path"  Testing 
Longer  Begets  Bigger 
Shooting  the  Messenger 
Feeding  the  Sacred  Cow 
Staff  Burnout  and  Turnover 
Robbing  Peter  to  Pay  Paul 

SEI  Proprietary;  Distribution:  Director’s  Office  Permission  Required 
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Contact 


William  E.  Novak 

Senior  Member  of  Engineering  Staff 
Military  Services  Team 
Acquisition  Support  Program 
Telephone:  412.268.5519 
Email:  wen@sei.cmu.edu 

Address 

Software  Engineering  Institute 
Carnegie  Mellon  University 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-3890 
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